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Apparatus and method for adaptively interpolating a full color image 
utilizing luminance gradients. 
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Abstract 



Adaptive interpolation is performed by apparatus operating upon a digitized image signal obtained 
from an image sensor having color photosites that generate a plurality of color values, but only one 
color per photosite. A digital processor obtains gradient values from the differences between 
luminance values in vertical and horizontal image directions. The gradient values are compared to a 
programmable threshold in order to select one of the directions as the preferred orientation for the 
interpolation of additional luminance v alues The interpolation is then performed upon values selected 



to agree with the preferred orientation. 
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(2) 

1 

J6B] 

immM 1 ] 1 fi^©^7-ffi$:MtSI 

&<£>*v-7* MM h&*-ra-f ^ — ^-b>it^e>f# 

^ % ^ < * L«cv^ J: o StttftStf * a = 

MM HaBC*H-SWiPW»K«fe. -tO^a-MM 10 
Mit©ffiB(D7t MM M4B©»ffiffi*^1fM-i-S 

Hul3L/t^ffii:©lt«tc/S^T, ii3§B'J>&< 2o© 

2 ] fjf^JI 1 E«©««fc:*5V^T^ 

ifi#ftT*fc£Chft^^irs^Mo 

3 ] m&m 1 K«0>S«fc:*5^-t\ 

^- > * tt ft d tr - 4: fc«r * £ * 6 & ■ . 
[0 0 0 1 ] 

[0 0 0 2] 

[tit#a>ttffl5] Hi— 0>>£OU) Hr 7 BU C 

• — > : regular pattern ) £WJS^&HMfcfiMK£ 

^sffliHim«c»brsite3i«ft*m-rs. 40 

(C F A) IE«/**->K«eoT. a£<bf StT^-b^l/ttt 
t^-te;i/£*tUT 1 fe<D^ftJi$rr£o z<Dtz&>, &4 
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2 

^Scf s fc «>£ ttmiH^f&R t a s . 

[0 0 0 3] — flaWfe**^ Sxy^AJiUy h\ yj- 

7J^*<JXAit »ffiWKHt>^U >P£titz (sparse 
ly sampled) #^-M/— (t:^t;i/ri:l:i6) 
ft, i^r\ yj-x T^u— (RGB) (Z)^;i/^^ — 

C^>7U >^^n^rS/W)^ (high-frequency) {f 

e>©;W'J-7 (bilinear : ZSWftJB) ffiRBtfffl^ 6ft 
UfcyU-> (missing green ) j fitftWF^"f £> fctf) 

Ke^n^fcli^n*— t:^-fe;i/ (oat) r ^ 

cD^'J-^-^ftffl^So Jfc^T-* feld^^ 

*^>*^ftt£#CDM-f 'J^rffiP^T-ffiPtSU 
& tr ^ ± ) v iz O l ^ T C F A * — >CD flb CO fe CD M P^ ft fr 

oc cc7)cto^^e>^^^^ov^T(i. *mm$m 

4, 6 4 2, 6 7 8^£g85*$iVt^£, 
[0 0 0 4] 

—"r^yrZY (artifact) CI i:T-fe5o CCD 

H«tt. «JiLK*H«rWSlS4, 6 3 0, 3 0 7^tBH^ 

©T-^Sc CCD 4, 6 3 0, 3 0 7§*H$rFTI±, i£ 
Rffitt (neiborhood) tc#ffi-rs!f#® (7-f— ^i*) C 
ov^TCD#HU^ftWffl1-So CCD-T ^— yf-^i 
V^aWftT;!/^ 'JXA^^tl. CCDStR^n^r 

CCD 4, 6 3 0, 3 0 7^I^T(t 

[0 0 0 5] -r-cc»«bfc«fc-5fc:, ±iase*^i£tcj: 

t>7 rtf«5fe*rsci:Tffti. ±134, 6 3 0, 3 0 7 
**H»*lF<fcfc:©»J5S:^«K:«t5, ccdt-t^^^t 

[o o o 6] *«w©i«ti^ ^miz^x hftJi^c^^ 

Sfi)7-f>f77^hSEiffiU H^cDP0^Sftfp]± 
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[0 0 0 7] ££1-, *ftW<DBmt* jj^—x-vW 
7-T-f77?hWMt, mm.(D; -Y X$:(SMU, 
IBfc©l!fBEfci««>Sfc«>fc, «*TfH*Sfl«j (7^7r^ 

[0 0 0 8] 

©-Y pt— i?-tr>+M±jS»CD*v- • 7^- MM h (phot 10 
osite ) wn?.Ii©*7--7*h^ h(i 

*»©*^— hottac 

iSV>T«U fc£ 1 0©*7-fit©^££fi£-f So C©<fc 
-5 tte^MtSSSBtt, a) r-fy^M^tifc'f^-^ 
fS^&x JSJSST^P ^>X©Hf£fft (^^—^•'7 
-Y X : image-wise) — > (fc£ U ;0;^->t; 
*5V>Tx ^<o*©7* MM M4M* s *fSfii£l$fe& 
V>) tlsTWm-tzm^Wm^&it, b) iftgB@-<fStt 

iDW&»Kfla*x Kft«©ffiP©7* h 20 

[0 0 0 9] *fSWlcJ:ixa, BUS3'7"o-t: >vit 

a) '>&< hfc 2o©-<y-s;*|qit*srtaBPKfi 
P^©M(;J;!3^ffi C^tw^ h • «'Ja- : grad 
ient values ) *4*t4*Sk, b) HulBPg^ffifii^ 
'J>&<i:*> loObfctMlfcJfcit-i- 3 c) huIB 

U*VMBi:©itttfcJ6lJT, tti>nWfc»Ktt©*IH©& 
tocifatt^'ip] (^u^y-T 1 — >> 3 >) tut, mitB'> 

&<£&2o©f^— S?7j|S|©5t> 1 oSrMJR-rs^lft 30 
©»Sf6£WP^bTx 8!lEWidfl9ffS«ft*«)**S 
[0 0 10] *#6W©5E«t»*B:, J&SrP-tr^SrSllJg 

[0 0 11] 

[Slflfcfll] A»v-7-f W7U-f &rav>fc#— • te>1t© 
S?#^fcWl*a©fc©&©T% .LvLT-fciu *f6^Kl^ 
S^a^^t£©-gB^^1-i)S^v *4V>tt*f5W 40 

t«4«Baw*i£fc«t»)ia»«>iiitt«-i-*5*©*& 

bT«fc<^. 

[0 0 12] Hl&VBI2t:-3i^TlftWrs. S 

fWxf^ • * ;>< 7 ttil^A^gC 2 hMP^ • fa&ss 4 t 
TW^ftSo A73g|52li. drX^* h : su bjec 
t ) (H^T) *»e.©-f *—V 1 )£*.4 * — i?-k>V 1 
2£jg<fcto©gy£g|5 1 O^^tro E^£*lT^ft^ 

c©ft^gciottx ^•j^M&aiM-rs'f ^— 50 



to©i>-V.y*£^/uT-^5,> aS^Eti^T^— ^ 
* MM r©2&7G7*U"r £^tft?>-tf 1 2 (i, J^]*P© 

- A#sstt«© ^ tn*s ffl u fcce*©*^^*^ 

(CCD) T*fe5o -te>+r 1 2fcJu ;Wt- ^V-f 
(Bayer array) £ bT*fl £.ft?> # ^ — 7 JI/^71/^ 
( C F A) 1 3 ic .fcoTgfrftTv-^o ;cD^t-- 

TWConTtt, 3Ra^tfF^3, 9 7 1, 0 6 5^1: 

>+r© lo©7* MM K i-&fc)-%WmZf}^— LT 

#tf:7\>l/-) ^i'y*-7P-K**CiaB$nfc» 
g#5>- (yj->) ©|H£1»abT^a©#1$»fl<JT? 

-tr>1M 2*«-f pt — S?3teftS3te-T4ii:CJ:t), 
7tD^/- i^S^ttlS (^ + — • f >73t-^— > 
a > : charge information) * s &^©7^ MM M3:fc 

^Tsasns. £©«««*»*, w-f*-Hi4 

CtftS&^tu &fc^>f F l 4ttc:©«*StiHllft&'<? 
©7tD^/-yi?liA/D3>/^ 1 6Cil& 

li»iT-f/-yn'v7 7 1 8lzm^^ti^>o -Y * — "J 
M?r77l8tt, a»©#±-r ^-^^iBit-r 

2>©l3<£>S^:^fi$:^-r?>^>^A7'^-trX^ ; &'J (R 
AM) T*fe?>o 

[0 0 1 3] 7 , P-tr^+f2 Oii, {M.%^> 1 0©JKt>^ 

TSttttg^SrSiJffllbv -t>1r 1 2©igft J f>-t>tH 
2 % © -Y p< — >>ti?8© ^ p >v * > ^© to £ £>S^7K 

{s^-tz^^ > McfcTU -f^-^7 7 7 1 8h^ajb 

^©A7JgB2$:$iJW-rS (®m, M»7D* 0 

■b^it^^-ei-\s) . ^^©t^-y i'^jb'f -^ft 

Wpi-^^77l8CS«i?h5i:, «*$*ifc 
ft^ti^^ y*fl/*y^7DtspU- 2 2£«S&jSn 
5o f Y-^;^Wl'7 , DtvH7 2 2lt *P<^©W 
P^ •IB®gP4©x;i/-> r -y F50 ! Sl'-b^®J»t-?»o 7" 
P-t^+f 2 2«7 r -f i^W^-^fg^ttbTfctP^T 
;i/ifiJXA*#*, MF^^nfcfS^^n^^^ 2 6 
bt, ^©m'J^bpJtg'&^^UA-H 2 4tj»4. 

[0014] mm. &v£ftcH*-r*M«tt«fflftft 

XA©4J^iiigmA s 7"D-tX^'>y7 7' 2 8lclBH$ni. 
(307Dt7lM>y77 2 8l± v >f ^-$/M.y7 7 1 8 
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J«Si!iSt?ifeK»t:'f^-^'>7T 1 8 ^3^S^,^T 

&m£.im ye^©su±, *©«ra©iiSRfc:flc# 
7 u-a 5 ^ p< -ye^ojEe < t £ 

trff^C^D v*#»*>ii&ttfttt&£*e^. tot, 
lifcA^wig^ M«7l 8 t«&g&H5fi©:/D 

[0 0 15] A7}SB2t±, ii^©*^ ^©a-^W— ~>3 

>co»>$5u-r (mm) -cmbiri. asma 10 

<fl-*«*ftfct>©T-«i:^. Sftgpi Ofcfc, «3teH*t*B 
DT> tfijxfcfc' 1 00 0^©1 8>j&»€>R8>|H|*T©»IIH, 
•fe>+rl 2ft>T^-y3tt:«3te-r4. ifc^T?, 3©^ 

— ywffttx -fe>-9-i 2©7* mm r*»e>»&m$ 

— r-©fifk ^-^-v77l8Cf^in5. ~© 

2 0 1 2, A/DD>;t-^16, SVMy 20 

77l8tfltt6*h5B«it©«iljlbl/-Mi. ± 

ili©«fc-5J&:U — hT-©-< * — ^©if£j££jifiE1-& «fc-5 £ 

ti-r^ ^^;i/»>y^-;UT , D-fe>v^2 2fflUgl:<t 

[0 0 16] £©*Hifc:,fcS l-3©a*UVMB*fcLt 

^^-7"f ri&g©fc#>«fc»)*K trU5HMfcftH© 
*Lffl©fc«>fc»iR?- S - i: #T? * 3 £ fctffclf 

&*>3A,v dODfetot, ^-rstr^^-vwp^fcjiM* 5 30 

fe-5o z<Dtz&>, mifcTnZti&J A— ^*77 1 8 
— tstac^&ai-f ^-^^t-§ 40 

[0 0 17] «ft&i%883 Ottm-b 0£&tt£ 
ft, ^^©fcflsfcWffl&flHR*****. -©£?& 
W^gicte, ->-v y^-i&K, T/s-^*, l)t>W7 

ft) s 7-f-;i/f-/7 1/-A, (g^w-^u (My^UflS 
T) , (ftS^FS) , BJte^-K (T^-^ffift 
fcantti'* &£©— KSJfc^IlT*— 

©««***** ft*. ^t'J*-F24tt,l 50 
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6 

©R©5t>©v>rft*» *fte>5R#) *^rti 

[0 0 18] =r-i i?*)Vi/if-}-)l7zHzyV-2 2tt, -< 

SH2twSn5«IHtt«K:t£oT»IH|-rs. 

^{ftgtjb'ttS^^^— (5. • =r—$ • ^ 

!ja- : missing data values ) ©iflUjlill] 2 iZmZtl 
4^-*>;*fcf£oTifc©«fcdfcfTfoftS. £1\ 
'J— >^?g (missing green ) j ©tl-J'-fe^ (oj 0 „ 

(high frequency information) „ "3~&2o*>^' l J — >© 

«$a> annate: j: o-c^ft&n, *?js©«s (u>^ 

>>3> : rendition ) #3t#£ft£. fe^tf$g* s 

asw^tee <i-*efo-feyy->©ttt«) t43o-ceiR*j 

U-TTjiS^iSffiR^i 9 3fc«>?>ft, CF AM 
*->C:fcit?>{fe©#7-a s £fi£$ft6o H2(C^$tl 

«r5«>^7 l A©ttt6««a<bf WJKSi53 6 

©^ 7Jc¥7^lSISl>*Sia7D[S|©^©^ , ;— >t^-fe;u(a 

■C«t»)ft«*ii*»WK (*3E : ^r-fi> h:gra 
dient ) f"S-3£ N 7K^P^lS!^4>. Sfi*|fijt*», Sfc 

2*7C^iejC4>jto6«jt:*||BSn5. 

[0 0 19] d©^^ iri7D-b7v©^l©7vf^ 
7 r D«v^4 0C^^n?.<t^ic, 'J>&<i:&2o 

CE«*nfcti'Hr;UffiBt:*»tS»Kfla©56fe^tf 

i>5. *fjg (y;->) e^-bjuo^avsiaPBWK 

mK2o©;&fa*»c,»iRS:h,5. c©#Hi&*|qHc 
-af?.J:dt«¥JSfa* s 3ltR$tL, 4 4(ZJ5V> 

U*v^«WT©BMB («3E) ©l>-r*X7>tl?&ofc 2 

o©j?jg (y;->) e*-fe;i/tfc«A»e>» *t«ffi«*«5p 

(2^©73(Rl©^** s il73fc4 ) . StR^ti 

fctiS©»JSfit#, BW8««USv^ttTTft5J:-5«t* 
|fillz?&^fc2o©JWJS (yj-» K^Hr;i/©3p*at«fc 

43V^T^ *MBS*lfc$ n J->tt&x ^©ttg©*^©* 

6) , 8(c43^-CfeM* s »i±J^n?.o 

C ^/D '^7 4 8 {C^jV>Tv 2^/W 'J-TffiF^Srffl 



(5) 
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^ X g-WS t: 1 -te ;Hia©fiM^- * 

-*>b (RGB) tSffifiXUT^ct^L, ifettfeM 
If t LXU& L £ }c£!13I£?to T * «fc V>. 
[0 0 2 0] ^U->^-*fr^?lLfct^-te;M4g 

& y -f X u ^ ; u £ # 1- s x t- a H & & ^§3 {£ ii m m. 

(MTF) £W-f 3-fe>-tf£jSbTS3iBT3C,i:* s T-§ 

[0021] ztit>(Dmmfewmiz.ii*)®;&znz4 * 

— iSt—Zcd 4 o© * 7 1 rf U — (iWTOji t)T'fe5. 
[0 0 2 2] A. (7X¥Pg^ > L&^ffi) fro (S 

^ • :r -f t 1 — A- : high scene spatial detail ) 5:^ 20 

>Z'$>2>Z£Zm.\sTl^2><, C©*§"&> >— >tZ&^X 
f%tz^>tzffim. (X : structure ) fr&l^fe 

4o©£?-t,;i/fr¥^b£:fi&,, 
[0 0 2 3] B. (*¥PBfll g Lg^ffiO fro (S. 
fflBM > LSVM*) ©t§^ SMHtttiU *¥ 

#|n]fr:£El$fc->->1ijg£*-ro ^->©*t?IHCi£t£ 
Lv ^->©:ty ^£«fcoT£C3g*M£g'Mb-3-3fc 

tote 7mxmz¥-t%ih-f&ztiz.±t)m&ftt>ti 

So £5>K, *>— >©*»««#B!«fe "fr e>¥-t$ib 3ft 30 

fc" fe©T*^^©T-s m&ommz (->*-x*x) fr 

g*{b;*ft£. 

[0 0 2 4] C. (zMZfgli > u^fit) fro (a 
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8 

C, »MttaiS#ra©¥*SfbC.fc9fTfc>fta. £<bJc s 

^©T% Bffe©lSW;*#«*<b«*ftS. 
[0 0 2 5] D . (^^Hffli ^ UgWtt) fro (S 

->mmifiiziiA,z:te^o 4o©f^-tr;i/frJpi£i{b$ 
fr{S?)S£ft&o -©£«>, lfffl777h Ji/h* 

«*fc*J^TH< <=>fr© Ty^Xj fSMfrfrfrftSo 
[0 0 2 6] «fS©CFA/^->t:|S^$nat){tTf 
t±&ofr\ e©MI13ttffl5fc}:cft£T-. m3£^3ft5 J: 
■SfrRGB/W-V 7 — (Bayer ) • ji^> — y -i 7 U ^ 
fCiSffl^ftT^So C©/W-^— 7L"f © 1 0©!RfM 
tiu RGB4*TBHc^-r«ta«:«f»W&^'f • Xo 

[0 0 2 7] 
[mi ] 

G R 
B G 

■ttat>%, 2o©«fjg (<7l>->) tffft loffl^M 
±tzfeO x 2o©£nv (i/y h'h^l/-) frft!2,#©*t 
ft«±t*S. »M©B^**©t;*^;MiBfc*rL 
ten?, 3fe£35£f#£C:£:t;:&.5„ 

[0 0 2 8] 1214 (A) ~ (D) fcttu «P^$*x5 lo 
©tT^-tr^ (jtLTH/wT-feS) feglt- 3 4 o©7 x — X 

(^T-HA,^gP^) • Xd 

&tr^-fe^l£^©*Th?iJ£;fclt.5teg 
t<t tliS^iJ^nSo =&7x-X(e43V^T, G (X, 

y) , r (x, v) , b (x, y) i:v^^*sBtt. mm 

Sftfc (ATHSsnfc) t^-tr;Ufrt>©7K¥3?[pl^{ji 
(X) /S^Sifi^lol^te (Y) lCOV>T^r ^-^-fe>it 

t«t-3TSJ^bfc«[Tf»S. Adap G(X,Y) tt, ti.T©7' 

3 • J X A £ ffl W > T ffi $ n fc 9 U - > fit X- % i> . 
[0 0 2 9] 
[&2] 
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9 10 
For Gdiff-hor = |G<-1,0) - 0(1,0)1 
Gdiff-ver = 10(0,-1) - 0(0,1)1 
Threshold = Predetermined value 
If (Gdiff-hor < Threshold) and (Gdiff-ver < Threshold) 

Or (Gdiff-hor > Threshold) and (Gdiff-ver > Threshold) 
Then lAdap G(X,Y) = [G(-1,0) + 0(1,0) + G(0,-1) + 
G<0, 1)3/41 

j 

If (Gdiff-hor < Threshold) and (Gdiff-ver > Threshold) 
Then lAdap G(X,Y) = [G<-1,0) + G(l,0)]/2| 

If (Gdiff-hor > Threshold) and (Gdiff-ver < Threshold) 
Then lAdap G(X,Y) = [0(0,-1) +0(0,1)1/21 



fcfiL--T% "Threshhold" tt ru&VMttj ft. "Pr 
edetermined value " r Pi?^fftj ftife~£~o 
[0 0 3 0] CfiD7;^'JXATtts £t\ Gdiff-hor 
ft, ffim$tiZ\ZP*)\'(D7k¥l5fft<D5£mv#*fr<D{r 20 
*)->m<om h£»U Gdiff-ver ft RID 

fB) i:^*UT^So f Ut, Gdiff-hor &t>*Gdiff-ve 

£)L§Wi£ *>'h£^»&«±, Adap G(X,Y) ft4o(2t 8 
d7^;i/CD^U— >«©¥J9i:-rSo £fc. Gdiff-hor # 



LS^tt«t5'J^<> Gdiff-ver # L £ WfiJ; D ^ 
»^HU Adap G(X,Y) ft7X¥^fRj©^U->fSO¥^i: 
LTt^o ttz, Gdiff-hor ifi L § Wfi J: t) < , Gd 
iff-ver #U£Wi£ ^'h£Oi§^yu Adap G(X,Y) ft 

[0031]EI4(A) — (D) 04o©7x-Xt:K 

^ 85 £ mm £ ti z> %m t )v =f u x a a o -e & 

[0 0 3 2] 
[»3] 
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11 



12 



7i-X 0 0 (M4 (A) ) 
G = G{0,0) 

B = 0.5 * [B(0,-1) - Adap G(0,-1) 

G(0,1) ] + G<0,0) 
R = 0.5 * [fc(-l.O) - Adap G{-1,0) 

G(1,0) ] + G(O,0> 



B { 0 , 1 ) - Adap 



R ( 1 , 0 ) - Adap 



7i-X 0 1 (H4 (B) ) 
R = R<0,0) 
G = Adap G ( 0 , 0 ) 

B = 0.25 * [B<-1.-1) - Adap G{-1,-1> + B(-l,l) 
Adap G(-l,l) + B<1,-1) - Adap G(l,-1) + 
B(l,l) - Adap G(l,l)] + Adap G(0,0) 

?*-x' 1 0 (M4 (C) ) 

B = B(0, 0) 

G = Adap GC0,0) 

R = 0.25 * [R{-1.-1) - Adap G(-l.-l) + R(-l.l) 
Adap G(-l,l> + R{1,-1) - Adap G{1,-1) + 
R(l,l) - Adap G(l,l)] + Adap G<0,0) 



1 1 (04 (D) ) 

G = G(0, 0) 

R = 0.5 * [R(0,-D - Adap G<0.-1) -t- R(0,1) 

G(0,l) ] + Adap G(0,0) 
B = 0.5 * [B(-1,0) - Adap G(-1,0) + Bd-O) 

G(l. 0) ] + Adap G(0. 0) 



- Adap 

- Adap 



-^A S ^LTV^2,>'3L-X (7i-X0 1M1 0) 

tv yj-> mm *«»j5TOt:*iiffl*ns. 
(at-ba,*:) ^^©^p&gfr^tofceM (r 

— G, B-G) StBLT/M'J^CiH^nS. & 

[0 0 3 3] *»w©*nma«tt« : ?*^^c*iv>Tffl 
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